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(54) Title: MULTIMEDIA ARCHITECTURE FOR INTERACTIVE ADVERTISING 



(57) Abstract 

An interactive multi-media system having a 
video server for providing a primary video program 
interleaved with secondary multimedia programming 
and a database for storing information about a viewer 
and the content of the primary video program. The 
secondary multimedia programming has a varying 
content determined by the data contained in the 
database, and as such, can be changed based upon an 
individual viewer's demographics in conjunction with 
the subject matter being displayed by the primary 
video program. Preferably, the system is accessed 
via the Internet, wherein the subscriber's house is 
connected to an Internet head end or node via a cable 
modem. 
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MULTIMEDIA ARCHITECTURE FOR INTERACTIVE ADVERTISING 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention is directed to a 
5 multimedia architecture for providing interactive 
video programming to an individual subscriber, for 
example to a television set, and, more particularly, 
directed to a multimedia architecture that provides 
on demand programming interleaved with secondary 
10 programming, such as advertising, to a subscriber 

wherein the content of the secondary programming is 
dependent upon information gathered about the 



viewer . 



15 Description of the Related Art 

Current interactive applications are 
typically limited to the providing of on-demand 
movies, and games, via existing cable systems and 
telephone systems. For example, Bell Atlantic is 
20 currently testing an on-demand movie system wherein 
subscribers are able to, for a fee, select a movie, 
from a list of movies. The selected movie is then 
transmitted to the subscribers television set. 

Currently, commercial messages, i.e. 
25 advertising, are delivered to a subscriber's TV from 
centralized broadcast facilities at either a 
national or local a level. The national and local 
providers simply fill-in designated, time-based, 
gaps in a primary video program with commercials. 
The commercials are, at best, picked based upon a 
demographic profile of a probable cross-section of 
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wherein information obtained from a viewer alters 
the flow of material to the viewer. 

It is yet another object of the present 
invention to provide on-demand primary programming 
5 with interactive advertising. 

It is a further object of the present 
invention to provide a primary video stream which is 
interleaved with a secondary video stream at a time 
in the primary video stream wherein the images 
10 displayed by the primary video stream are related to 
the content of the secondary video stream. 

It is yet another object of the present 
invention to provide a primary video stream having 
commercial message breaks which are dependent upon 
15 the content of the primary video stream and the 
demographics of an individual subscriber. 

Additional objects and advantages of the 
invention will be set forth in part in the 
description which follows, and, in part, would be 
20 obvious from the description, or may be learned by 
practice of the invention. 

These and other objects of the invention 
are achieved in an interactive multimedia system 
comprising video server means for providing a 
25 primary video program interleaved with secondary 

programming, the video server means being adapted to 
receive information from a viewer during display of 
the primary video programming and the secondary 
program, and database means for storing the 
30 information about a viewer and analyzing the 

information to generate viewer data which causes the 
video server means to vary the content of the 
secondary programming displayed to the viewer based 
on the viewer data. 
35 Objects of the present invention are also 

achieved in a method of gathering data from a viewer 
comprising displaying a primary video program, 
interleaving a secondary multimedia programming with 
the display of the primary video program, presenting 
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BRIEF DESCRIPTION OF THE DRAWINGS 

encs ' taken in conjunction with t-h- 
accompanying drawings of which: 

Fig. ! is a block d . agram Qf ^ 

xnteractxve advertising system in accordance with a 
f-st preferred embodiment of the present invent^. 
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Fig. 2 is a block diagram of a video 
server in accordance with the first preferred 
embodiment of the present invention. 

Fig. 3 is a broad band processing unit for 
5 use in the video server in accordance with the first 
preferred of the present invention. 

Fig. 4 is a block diagram of an 
interactive advertising system in accordance with a 
second preferred embodiment of the present 
10 invention. 

Fig. 5 is a block diagram of a set -top 
unit in accordance with the preferred embodiments of 
the present invention. 

Fig. 6 is a flow chart of the top level 
15 process for the video server in accordance with 

either the first or second preferred embodiments of 
the present invention. 

Fig. 7 is a flow chart of a receiving 
process in the video server in accordance with the 
20 preferred embodiments of the present invention. 

Fig. 8 is a flow chart of the send video 
data process used in the video server in accordance 
with the preferred embodiments of the present 
invention. 

2 5 Fig. 9 is a diagram of an MPEG stream as 

used in the preferred embodiment of the present 
invention. 

Fig. 10 is a block diagram of an MPEG 
stream selector and multiplexer as used within the 
30 video processor in accordance with the preferred 

embodiments of the present invention. 

Fig. 11 is a flow chart of an interleaving 
process in accordance with the preferred embodiments 
of the present invention. 



35 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Reference will now be made in detail to 
the present preferred embodiments of the present 
invention, examples of which are illustrated in the 
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accompanying drawings, wherein like reference 
numerals refer to lilte elemente 

interactive advertising involves 

■■ slT't? 9 aPrimary Vide ° stream, ,or 

(or Zi the „ Primary Pr09ran,, bein * sh °»" to a viewer 
(or subscriber, generally comprising movies or 
educational material at- = j 

riai ' at a Predetermined time w-it-h 

simply, the secondary programs) , typically 
comprising advertising, educational or other 
commercial material. Additional facilities are 
prided to overlay textual and/or graphical 
material on the display and to receive input from 
the viewer. Thus, for example, the secondary 
Programming stream may he used to interactive^ 
request information from the viewer, such as 
demography and shopping preferences. The 
information obtained from the viewer may be utilized 

r oTd^rr ? ondaty — 

Ui display at a future tim^ • . 

:r ay °v he ' ><~T*zz:: 

the responses from the viewer and/,,,- -k 

the primary program. ' C ° ntent ° f 

to view as'theT"* 1 '' * Tl ~ r * 

View as the primary program, he can be prompted 

to select whether he wishes to view the mo^ie, or 

fuU charge without secondary programs, i.e 

advertising, the same movie for half charge with 

some secondary programming presented during the 

ZTsllZ" 9 " 9 ^-active advertising and 

questions, requiring a response from the viewer 
presented during the movie. The nature of the ' 

vaT^ ary , Pr ° 9ra ™ in9 PreSe " Ced '° tte *—r can 
vary baS ed on a number of determining factors, 
including the movie currently selected, a past 
history of the movies selected, the scene L the 
current movie at which the advertising will b e 
displayed, previously collected demographic 
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information about the viewer. In the case of 
interactive advertising, the advertising can change 
based upon the viewer's response to the questions 
presented during the interactive advertising in 
5 addition to any of the previous mentioned 
determining factors. 

Thus, for example, if a viewer selects an 
action movie as the primary program, if at some 
point a truck is highlighted in a scene, secondary 
10 programming material can be interleaved which 

presents an advertisement related to a local truck 
dealer carrying a similar model of truck being shown 
in the primary programming. Alternatively, in an 
educational setting, the secondary programming can 
15 be tailored to a viewer's level of knowledge. For 

example, if a viewer has a low technical skill level 
(as determined by prior interactive sessions) , the 
primary programming can be interleaved with 
secondary programming matter explaining technical 
20 concepts presented in the primary program. 

Alternatively, if the viewer has a high level of 
technological awareness, the primary program would 
be interleaved only with minimal secondary 
programming material . 
25 Alternatively, the secondary programming 

can be an interactive presentation requesting input 
from the viewer in the above examples. For example, 
the viewer could be prompted to indicate whether, in 
fact, the viewer desires to see the secondary 
30 programming. Alternatively, the viewer can be 

requested to answer questions about the material 
seen in the primary program to determine the amount 
of secondary programming needed. This feature would 
be especially useful in the educational environment . 
35 Any information collected during the secondary 

programming sessions can be stored in a database and 
used in future sessions to further refine the 
content of the secondary programming. The preferred 
embodiments are described as an interactive 



WO 97/12486 

PCT/US96/15465 

- 8 - 

advertising system, however the uses for such as 

III* t ^ """^ " >»t may be 

used for exampie. £or education and entertainment 

described as being videos, they may al so 
comprise animations, graphics, text, audio or any 
other multimedia medium. 

The primary program and secondary D roor,„ 
are^referabiy encoded in the » EOl formal ^ 
MPEG1 format requires a throughput averaging l 5 
megabits per second to provide a fair quality 
Picture. An E1 or Tl „ iring ^ Y 

adapted to handle such throughput. If higher 
quality V i de o is desired, an ^ encoding can be 
used. However. MPEG2 encoding requires a nigher 
average throughput, typically in excess of 4 
megabits per second, and as such would require an 
fl'u™* " S A ™ °' ~~ broach 

FIG. 1 is a block diagram of an 

first preferred embodiment of the present invention 
n accordance with the first preferred embodiment 
the prrmary and secondary programming material is 
delivered to a subscriber's house 500 via the 
Internet 530. 

with „ . E ! Ch SUbSCriber ' s house S00 is provided 
with a standard known TV 5 0 2 which is attached to a 
set-top unit SCO, described hereinafter. The set- 

e H III tl COnMCted " a 

An off-the-shelf P c *op u 

u 508 can be optionallv 

provi ed The set-top unit S04 and the optional' 

advert 1 " the -'"active 

advertising system. A cable modem 506 is typically 
connected to a head end 51 o via an existing " 
subscriber based cable network. The head end 5X0 
comprises a cable modem Sl 2 connected to the cable 
mode. S0 6 and the subscriber's house S00. 
modem Sla is connected to a cable router „« for 



delivering individualized programming material to 
each subscriber's house 500. 

The head end 510 is connected to the 
internet 530. Also connected to the Internet are a 
series of video servers 520a-520c. Each video 
server 520, discussed hereinafter, contains a 
content and subscriber database 522 and a 
programming database 524. The programming database 
524 stores the primary and secondary programming. 
The content and subscriber database 522 stores 
statistical information regarding each of the 
subscribers, including demographics, network address 
and other related information. Additionally, the 
content and subscriber database 22 stores content 
information regarding the primary and secondary 
programming stored in the programming database 524. 
For example, with respect to video movies, the 
content and subscriber database 522 would store 
timing and scene information regarding each movie so 
as to enable a correlation between a content of a 
scene and the subscriber's demographic data. 

A standard Web server 540 is also 
connected to the Internet 530. The Web server 540 
serves out Web pages in HTML format to the Internet 
530 in what is called the World Wide Web. The Web 
server 540 is connected to a subscriber database 542 
which is typically a table relating a subscriber's 
IP address with his cable router address. 

In use, a subscriber will access the Web 
server 540 using a standard Web Browser via an 
optional personal computer 508 or the set-top unit 
504, which in this configuration would preferably 
comprise an Internet ready device. Using the Web 
Browser, the subscriber selects a primary program, 
e.g. a movie from the Web pages displayed by the 
Web server 40, which he wishes to view on the TV 
502. The Web server 540 accesses the subscriber 
database 542 to resolve the subscriber's address. 
Thereafter, the Web server 540 directs the 
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appropriate video server 520 to begin delivery of 
the primary program. The appropriate video server 
520 can be selected based upon geographic location 
the location of the subscriber's information, or 
even the location of the programming material 
Thereafter, the video server 520 delivers the 
primary and secondary program material to the cable 
router 514 and the head end 510 via the Internet 
510. The cable router 514 routes the program 
material to the appropriate subscriber's house 500 
As the set-top unit 504 provides a two way 
communication route with the video server 520, an 
interactive viewing session can be carried out, as 
discussed hereinafter. 

While the link to the set -top unit 504 
from the head end 510 has been described as a 
standard cable base subscriber system, one of 
ordinary skill in the art will recognize that an 
asymmetric digital subscriber link may also be 
20 utilized. Further, various additional links such as 
a Tl or an El data line may also be utilized. 

FIG. 2 is a block diagram of a video 
server 520 in accordance with the first preferred 
embodiment of the present invention. Specifically, 
FIG. 2 sets forth a video server 520 which can form 
an interactive advertising system for use in 
outputting signals to a broad band public network 
for example, ATM, OC12, etc or a broadcast cable 
system as used in the preferred embodiment. This 
system is particularly adapted to providing 
programming over existing cable-TV networks 
utilizing standard co-axial cabling to each 
subscriber's home. 

In accordance with the preferred 
embodiments of the present invention, the video 
server 520 may be realized on a modular multimedia 
architecture which utilizes multiple application 
processing units (hereinafter APU) . United States 
Patent No. 5,029,199 and United States Patent No 



25 



30 



5,193,110, incorporated herein by references, 
describe a distributed control and storage systems 
which may be modified, in accordance with the 
present invention to produce a system for providing 
interactive advertising. In accordance with the 
first preferred embodiment of the present invention 
APUs 10 and 12 are utilized to handle voice and 
application data. Additionally, three broad band 
processing units, 14, 16 and 18 (hereinafter BPU, ) 
are utilized to handle video applications and video 
data, which are outputted to a router 50. The 
router 50 interfaces with the Internet 530 (or 
directly to a broad band network) . As shown in 
Figure 1, APU 10 and APU 12 share a SCSI disc 20. 
The APU 12 has access to a SCSI 22. Likewise BPU 
14, BPU 16 and BPU 18 each have access to SCSI discs 
24, 26 and 28, respectively which operate as the 
programming database 524 and holds video programming 
including movies and advertisement. 

The APUs 10 and 12 are connected to a 
narrowband digital switching matrix 30 via Tl lines. 
The narrowband digital switching matrix 30 is 
connected to a narrowband private switched telephone 
network (hereinafter PSTN) 46 via El lines 44. A 
master control unit controls the narrowband digital 
switching matrix 30 and the router 50. The master 
control unit 32 is connected to the APU 10, the APU 
12, the BPU 14, the BPU 16, the BPU 18 and the 
router 50 via an ethernet backbone 48. Preferably 
the master control unit 32 has an associated SCSI 
disc array 36. A database unit 38, for collecting 
and analyzing data from each of the various 
components, is also connected to the ethernet 
backbone 48. Preferably the database unit 38 is 
provided with a SCSI disk array 42. The database 
unit 38 acts as the content and subscriber database 
522 and stores various information about subscribers 
to the system including demographics, past viewing 
history/ P ast answers to interactive questions, and 
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other information as necessary. Additionally, the 
database unit 38 can be programmed to make decisions 
regarding when to interrupt a movie with advertising 
and what advertising programming to use. However, 
5 the BPUs can also be programmed for this task. 

An multiple integration unit 52 
(hereinafter MIU) is connected to the narrowband 
PSTN 46 using an SMDI (simplified message desk 
interface) protocol. The MIU 52 is also connected 
10 to the ethemet backbone 48. The MIU 52 interfaces 

with other telephone systems, such as PBXS. A call 
control adjunct (CCA) 54 is connected to narrowband 
digital switching matrix 30 and to the narrowband 
PSTN 46 via the common channel system signaling 7 
15 protocol (hereinafter CSS 7) . The CCA 54 is also 

connected to the ethernet backbone 48. A network 
interface unit 56 (hereinafter NIU) is connected to 
the ethernet 48 and interfaces to other systems via 
FDDI (fiber distributed data interface) . 

As shown in Figure 2, the master control 
unit 32, the database unit 38, the narrowband 
switching matrix 30, the MIU 52, the CCA 54, the NIU 
56 and the router 50 are each preferably provided 
with a backup redundancy unit. 

The MCU 32 acts to assign an incoming call 
Un the case of voice mail, which can also include 
requests for video programming) , via the narrow band 
PSTN 46. The MCU32 directs requests for video 
programming, via the router 50 or the narrow band 
PSTN 46, to an appropriate APU or BPU over the 
ethernet backbone 48. Control data messages between 
the MCU 32 and the APUs or BPUs, or between the APUs 
and the BPUs, is routed over the ethernet backbone 
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48, 



During operation, a request for voice or 
video services is issued via the narrowband PSTN 46 
or the Internet via the router 50. If the services 
are voice services, such as voice mail, the master 
control unit 32 routes the request to an appropriate 
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APU for service of the request by an APU via the 
narrowband digital switching matrix 30. If the 
request is for video services, the master control 32 
routes the request, via the ethernet backbone 48, to 
the appropriate BPU which provides services via the 
router 50, but may, with the addition of an 
appropriate link, be output via the in narrow band 
digital switching matrix 30. 

In general, the video programming provided 
by the BPUs may be either normal 
entertainment/educational programming or 
advertisement programming. The BPUs may access the 
database unit 38 to determine what type of 
programming to provide to a subscriber. The 
programming may be divided among the BPUs or 
arranged dynamically. Further, a single massive 
disk array may be used and the requests for services 
allocated across the BPUs based on the number of 
requests . 

FIG. 3 is a block diagram of a BPU 14 for 
use in the video server 520 in accordance with the 
first preferred embodiment of the present invention. 
The BPU 14 is provided with a CPU 64 which 
interfaces with the ethernet backbone 48 via lOmbps 
ethernet adapters 66 and 68. The CPU 64 outputs a 
signal via lOOmbps ethernet adapters 70 and 72 to 
the router 50. The BPU 14 is provided with an 
internal memory 74. An array of SCSI adapters 76 
interface with an associated array of SCSI discs 78 
which preferably store the primary and secondary 
programming material. The CPU 64 can, for example, 
comprise a 200 megahertz peutium processor. In 
addition, each BPU can, for example, comprise a 
standard PC mother board provided with the 
appropriate 10 cards. 

Although the first preferred embodiment of 
the present invention has been described with 
respect to a specific configuration, will be 
recognized at the first preferred embodiment is not 
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limited to the specific configuration. For example, 
while the present system has been described as 
outputting video signals to a router 50 for 
transmission over the Internet 53, the router 50 can 
> transmit information directly to the head end 510 
via a broad band PSTN or the narrow band PSTN 46. 
However, the narrow band PSTN 46 would have to 
provide an appropriate signal throughput for 
handling the video data. 

PIG. 4 is a block diagram of an 
interactive advertising system in accordance with a 
second preferred embodiment to the present 
invention. An APU 80 and an APU 82 interface via an 
ethernet network with bridge/CSU units 88 and 90, 
which in turn interface with a digital switching' 
system 96 (hereinafter DSS) . Similarly, an APU 84 
and an APU 86 interface with a DSS 96 via bridge/CSU 
units 92 and 94. The bridge/CSU units 88, 90, 92 
and 94 preferably output the via El lines. The 
output of the DSS 96 is converted back into a 
ethernet signal via CSU/bridge units 98, 100, 102 
and 104. Set-top units 504a-504d interface with the 
CSU/bridge units 98, 100, 102 and 104, respectively. 
The set-top units 504a-504d facilitate interaction 
25 between the APUs and the subscriber via TVs 502a- 

502d. The set-top unit 504a is further connected to 
a video conferencing unit 116, which provides video 
conferencing services via the TV 502a. The set-top 
unit 504a is connected to the TV 502a via an A/B 
switch 118, which also interfaces with the video 
conferencing unit 116. An alternative line with an 
amp 120 is provided for signal clarity. 

Figure 5 is a block diagram of a set -top 
unit 504 for use in an interactive advertising 
system in accordance with the preferred embodiments 
of the present invention. Preferably, the set- top 
unit 504 is configured around general purpose 
computer, i.e. a standard PC. A CPU 122 interfaces 
with the ethernet interface 126 for communication 
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with the APUs. An I/O unit 132 allows the viewer to 
interact with the set -top unit 504 via number of 
inputs, such as a keyboard, mouse, or an infrared 
remote control . An NTSC interface card 128 provides 
a signal to the TV 502. Additionally, an overlay 
processing unit 130 enables the CPU 122 to overlay 
text and graphics on top of a video stream. A 
memory 124 is connected to the CPU 122. 

The overlay processing unit 130 can be 
used to display texts and graphics to a viewer in 
conjunction with, or independently of the primary or 
secondary programming currently being displayed. 
For example, a list of movies could be displayed 
overtop a primary program showing a continuous 
-stream of trailers showing each movie. Thereafter, 
when a viewer selects a certain movie, such as with 
an inferred remote control unit, a keyboard or a 
mouse, the set-top unit 504 requests, from an 
assigned video server 520, a secondary program 
showing a longer trailer regarding the selected 
movie while the overlay processing unit displays a 
message requesting the viewer to input whether he 
wishes to receive the movie for free, with a lot of 
advertising, for half the price with minimal 
advertising, or for full price with no advertising. 
Further, the overlay processing unit can be used 
during the advertising programming segments to take 
an order for the advertised product, request input 
from the viewer regarding the viewers responses to 
the advertising, the desirability for future 
advertising regarding the product shown during the 
advertising radio programming, or other demographic 
information. Additionally, the overlay processing 
unit 130 can be adapted to display other services, 
such as e-mail, video mail, voice mail (within an 
appropriate speech to text translator), faxes, etc. 

In use, a viewer interacts with a set-top 
unit to request video services, i.e. primary and 
secondary programming, from the BPUs. The BPUs 
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deliver video services through the router 50 and the 
Internet 530 (in the first preferred embodiment) or 
the digital switching service 96 (in the second 
preferred embodiment) to the appropriate set -top 
5 unit 504, which displays the output on a TV 114. 

The overlay processing unit 13 0 enables the display 
of menus and other messages to the subscriber 
displaying information or prompting for input. 

FIG. 6 is a flow chart of the top level 
10 process for the video server in accordance with 

either the first or second preferred embodiment of 
the present invention. For the sake of clarity, the 
processor will be described with respect to the 
first preferred embodiment. The process starts in 
15 step 1000. Thereafter, the video server 520 waits 

until a connection request is received in step 1002. 
The connection request can come via the Web server 
54 0 over the Internet 530 or directly from the 
subscriber's P.C. 508. Once a connection request is 
received, it is logged in step 1004, subsequently, a 
receiving process, see FIG. 7, is started in step 
1006. After the receiving process has completed the 
process returns to step 1002. 

Fig. 7 is a flow chart of a receiving 
25 process in the video server in accordance with the 

preferred embodiments of the present invention. The 
receiving process is started in step 2000. 
Thereafter, the video server 520 gets a message from 
the set-top unit 504 in step 2002. In step 2004, 
the message is reviewed to determine whether it 
contains a file name. If the message contains a 
file name, the requested video file is opened in 
step 2006. The process shown and described in Fig. 
7 is directed toward providing a primary video 
35 stream as the primary program and a secondary video 

stream as the secondary program, but other types of 
streams may be used such as simple graphics or 
audio. The team stream indicates that the data is 
sent and received as a linear, or non-linear, 
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sequence of data. The term will become more clear 
in reference to Fig. 9 which shows a typical MPEG 
stream which preferably constitutes the primary and 
secondary video streams. Thereafter, a send video 
data process is started, see Fig. 8, in step 2008. 
In step 2009, a secondary video program is 
interactively interleaved with the primary video 
programming. When the end of file of the primary 
video programming is reached, the connected is 
closed in step 210 and the receiving process is 
ended in step 2032 . 

If, in step 2004, the message does not 
contain a file name, it is deemed to be a command 
requesting a change in the flow of the primary video 
program, such as rewind, fast forward, pause, stop 
or the presentation of a secondary video program. 
In step 2012, the data file of the requesting set- 
top unit 504 is opened. Thereafter in step 2014 it 
is determined whether the message requires 
additional information, such that a question needs 
to be displayed on the TV 502. If a return question 
is to be displayed, an overlay is sent to the 
overlay processing unit 130 in the set -top unit 504 
in step 2016. Thereafter, a response is received in 
step 2018 and the process ends in step 2032. If the 
response can be handled by the set- top unit 504, 
such a process is initiated. However, if the 
response needs to be handled by the video server 
520, the response is treated as another message from 
the set -top unit and another receiving process is 
started with the response being entered as the 
message. 

If, in step 2014, the message does not 
require a return question, it is determined whether 
the viewer is viewing a -must view" video in step 
2020. A "must view" video may comprise, for 
example, advertising when the viewer has indicated 
that he wishes to receive advertising in return for 
a reduced cost of viewing the primary video 
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zzzz. a :::™-— . ^ nay 

a -rtain segj t of t he T™ " ^ that 
Alt.™.-- , £ ™ ent o£ the Population receives it. 
Alternatively. in an educational setting a .„„ 

view" tr-i^^^ o^^uj-ug, a "must 

view vl deo may comprise certain key points of the 

~?r iai which the stJi - ~ 

video ly, V1SWer 18 VieWin 9 a "«»t view" 

leti d e ~ rr eds to step 2022 -~ - - 

shift in th/ reqU6StS a fo ^rd 

tre am " Vid6 ° Stream ° r - - 
shift COmmand d ° eS * ^rward 

shaft or change from the current video stream th. 
process is waited until th» „ scream, the 

, ri , until the current "must view" 

video stream is completed in ste p 2024 

viewer is !o7 " determi <^ that the 

viewer ls not vxewmg a « raust view „ 

Process proceeds to step 2026 wherein th!' 
video stream is opened to th. requested 
the command At thTf re ^sted position in 

data is Tl P ° lnt ' the re ^«ted video 

data is sent in a sending process, see Pi g . 8 
Thereafter, in step 2028, secondary video 
Programming is interactively interleaved into the 

reached^ ^ Vide ° Programming is 

-or' S^tTy 1 ^" ~ ^ ™ " 
2032. 9u«"y. the process ends in step 

Fig. a is a now chare o£ 
data process used in th« v^.- 

win, «m the video server in accordance 

with the preferred embodiments of the present 
invention. Th e sending process starts in step 3000 
in step 3002 data is read from -p 3.». 

the set-top unit 504 via the Internet S30. 

the"" 7' in " eP 3004 ' " is ««• «—d whether 
the set-top unit has closed the connection, if the 
set^top u„ lt =04 has not closed the connection, lt 

Of tl T in S " P 3006 •*•«« the end of ile 
of the data bein g sent has been reached. „ tta end 



of file has not been reached, the process continues 
to read the disk and send data in step 3002. Once 
the set-top has closed the connection in step 3004, 
or the end of file has been reached in step 3006, 
the process ends in step 3 008. 

The interleaving of secondary video 
programming with the primary video programming can 
be performed at time based intervals, or based upon 
the content of the primary video programming. 
Additionally, because of the presence of a return 
communication path from the set-top unit 504, the 
secondary video programming can also be contingent 
upon commands received from the set-top unit 504. 
Additionally, it is important to note, that the 
secondary video programming can also be considered 
primary video programming to further secondary video 
programming, thus in effect allowing a hierarchical 
tree of video programming to be presented to the 

subscriber . 

In general, in order to attain 
interleaving based upon the content of the primary 
video programming, the primary video programming 
must be analyzed prior to display, and a series of 
records generated setting forth the timing and 
subject matter of the material being displayed 
during the primary video programming. For example, 
various subject matters can be defined in 
relationship to the secondary video programming and 
assigned numbers which are then stored as a record 
with the respective timings in the primary video 

programming . 

Fig. 9 is a diagram of an MPEG stream 600 
as used in the preferred embodiment of the present 
invention. An MPEG stream 600 comprises a stream 
header 602 followed by a series of data packets 604a 
to 604n. The stream header 602 includes information 
on the time at which the first data packet in the 
stream 600 is to be played. In the preferred 
embodiments, the primary and secondary programs are 
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MPEG streams and are referred to as the primary 
video stream and the secondary video stream. 

Each data packet 604 comprises a group of 
Pictures (GOP, 610 and a series of frames, m 
general, there are three types of frames, a b-frame 

' 614 and a P-*«« 6«. The i- frame 

614 composes a full frame, while each of the fa- 
frame 612n and the p-frame 616n contain predictive 
changes to the full frame, contained in the i-frame 

Fig. io is a block diagram Qf gn MpEQ 
stream Elector and multiplexer 700 as used within 
the video processor in accordance with the preferred 
embodiments of the present invention. Preferably 
the MPEG stream selector and multiplexer 700 is 

HflT ll T vare in each Bpu and executed * 

CPU M. The „P EG stream selector mult±plexer 
700 is provided with a parameter buffer 702 
(preferably in the memory 74) and a subscriber 

parameters/MPEG stream content breaks database 704 
(preferably formed in the array of SCSI disks 78) 

The MPEG stream selector and multiplexer 700 

receives and multiplexes a MP*r _ 

p xes a MPEG content stream 710, 
i.e. the primary video stream, and MPEG commercial 
segments 720a-7 20 n, i. e . the secondary ^ 
A sing e MPEG output stream 730 is output by the 
MPEG st r eam selector and multiplexer 700. An input 
line 740 is provided to enable the BPU or set-top 
unit 504 to control the selection of the primary and 
secondary video streams in response to commands from 
the subscriber. 

Pig. 11 is a flow chart of an interleaving 
proce SS in accordance w . th ^^^^ embod . ments 

of the present invention. The process starts in 
step 4000. The header of the primary video stream 
xs copied into the parameter buffer in step 4 002 
Concurrently, the header of the primary video stream 
18 d6llVered to the output channel 730 in step 4004 
In step 4006, the next break point in the 
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primary video stream is determined. The break point 
is the point at which the secondary video stream 
should be interleaved with the primary video stream. 
This decision is made based upon information 

5 contained in the subscriber parameters/MPEG stream 
content breaks database 704 or in response to a 
command from the subscriber. Typically, the 
subscriber parameter /MPEG stream content breaks 
database 704 will be generated prior to the showing 

10 of the primary video stream for each individual 

subscriber, by correlating the demographic data 
contained in the content and subscriber database 522 
with timing and subject information which has been 
previously generated about the primary video 

15 programming. While the interleaving process is 

being shown and described as being a linear process, 
it will be recognized by one of ordinary skill in 
the art that parallel processing can be used, 
wherein the break points are determined concurrently 

20 with the displaying of the primary video stream. 

In step 4008, the primary video stream is 
sent to the output channel. In step 4010, it is 
determined whether the end of file of the primary 
video stream has been reached. If the end of file 

25 has been reached, the process ends in step 4024. 

If the end of file has not been reached, 
it is determined whether a break point has been 
reached in step 4012. If a break point has not been 
reached, the process returns to step 4008 and 

30 continues to send primary video stream to the output 

channel 730. 

However, if, in step 4012, a break point 
has been reached, the primary video stream is 
stopped in step 4014. Subsequently, in step 4016, a 
35 reset code is sent to the set-top unit. Thereafter, 
in step 4 018, a secondary video stream, e.g. the 
MPEG commercial segment 720a-720b is sent to the 

output channel 730. 

Once the end of file of the secondary 
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video stream has been reached in step 4020, a new 
stream header 602 is generated in step 4022. A copy 
of this header is stored in the parameter buffer 

702. The new header contains new timing information 
regarding the playing of the remainder of the 
primary video stream. After the new header has been 
stored, in step 4022, the process returns to step 
4000 and the next break point is determined. 

As shown and described above, the 
preferred embodiments of the present invention 
enable an individualized video programming stream to 
be transmitted to a subscriber's house. The 
individualized video programming stream comprises a 
primary program, preferably formed of a primary 
video stream and a plurality of interleaved 
secondary programs, preferably formed of secondary 
video streams. The secondary video programs are 
interleaved, based upon input received from the 
viewer, subject matter being displayed in the 
primary material, and/or demographic and personal 
data regarding the subscriber stored in a database 
Additionally, as the system is interactive, the data 
regarding each viewer stored in the database can be 
updated in response to questions, by the overlay 
processing unit 130, presented during the 
presentation of either the primary or secondary 
material . While the system has been described with 
respect to an interactive advertising system, one of 
ordinary skill in the art will also recognize that 
it can be utilized in a variety of educational and 
entertainment scenarios . 

Although a few preferred embodiments of 
the present invention have been shown and described 
it would be appreciated by those skilled in the art' 
that changes may be made in these embodiments 
without departing from the principles and spirit of 
the invention, the scope of which is defined in the 
claims and their equivalents. 
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What is claimed is: 

1. An interactive multimedia system 

comprising: 

video server means for providing a primary 
5 video program interleaved with secondary 

programming, said video server means being adapted 
to receive information from a viewer during display 
of the primary video programming and the secondary 
programming ; and 

10 database means for storing the information 

about a viewer and analyzing the information to 
generate viewer data which causes said video server 
means to vary the content of the secondary 
programming displayed to the viewer based on the 

15 viewer data. 

2. An interactive Multimedia system, as 
set forth in claim 1, wherein the database means is 
adapted to determine the timing at which the 

20 secondary programming is displayed based upon a 

content of the primary video program. 

3. An interactive multimedia system, as 
set forth in claim 1, further comprising: 

25 interface means which receives the primary video 

program and the secondary programming from said 
video processing means and displays the video 
programming on a monitor to the viewer. 
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4. An interactive multimedia system, as 
set forth in claim 3, wherein said interface means 
as adapted to overlay the primary video program and 
secondary programming with graphics. 

5. An interactive multimedia system, as 
set forth in claim 3, wherein said interface means 
is adapted to gather information from the viewer and 

10 ^. t tranSmit thS Nation to said video processing 



15 



5 



6. An interactive multimedia system, as 
set forth in claim 1, wherein the primary video 
program is a video stream. 

7. An interactive multimedia system, as 
set forth in claim 6, wherein the primary video 
program is a movie. 

8. An interactive multimedia system, as 
set forth in claim l, wherein the secondary 
programming comprises advertisement. 

9- A method of gathering data from a 
viewer comprising: 

displaying a primary video program,- 
interleaving a secondary multimedia 

programming with the display of the primary video 

program; 

presenting questions to the viewer during 
at least one of the primary video program and the 
secondary multimedia programming; 

receiving answers to the questions from 
the viewer; and 

varying the content of the secondary 
multimedia programming based on the answers. 



10. A method of displaying comprising: 
displaying a primary video program having 



WO 97/12486 



- 25 - 



PCT/US96/15465 



a varying content; 

generating a video stream breaks file 
correlating the content of the primary video program 
with the timing at which the content is displayed; 
5 interleaving secondary multimedia 

programming with the display of the primary video 
program; 

varying the content of the secondary 
multimedia programming based on the video file 
10 stream breaks file. 

11. An interactive multimedia system 

comprising: 

subscriber database means for storing 
15 subscriber data regarding demographic and 

preferences of a subscriber; 

a video breaks database means which stores 
video breaks data correlating a content of a primary 
video program with a timing at which the content 
20 exists within the primary video program; and 

video server means for providing the 
primary video program to an individual viewer 
interleaved with secondary programming selected 
based on a correlation between the subscriber data 
and the video breaks data. 



WO 97/12486 



PCT/US96/1S465 



1 /11 




FIG. I 

SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



PCTAJS96/15465 




WO 97/12486 



PCTAJS96/15465 



3/11 




WO 97/12486 



PCTAJS96/1546S 




W ° 97/12486 PCT/US96U5465 

5/11 




SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



6/11 



PCT/US96/15465 




FIG. 6 



SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



PCT/US96/I5465 



7/11 

( START ) -\_^2000 



GET MESSAGE FROM 
SET-TOP UNIT 



2002 



OPEN THE REQUESTED 
VIDEO FILE 




OPEN THE 
REQUESTED VIDEO 
STREAM TO THE 
REQUESTED POSITION 
AND SEND VIDEO DATA 



INTERACTIVELY 
INTERLEAVE 
SECONDARY VIDEO 
PROGRAMMING 



I 



2028 



WHEN EOF IS REACHED 
UPDATE SET-TOP AND 
CLOSE THE CONNECTION 



WATT UNTIL CURRENT 
VIDEO STREAM IS 
COMPLETED 



2024 



^2030 



<E) 

C END V— -2032 



FIG. 7 

SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



8/11 



PCT/US96/15465 




( END X ^ 3008 



FIG. 8 



SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



PCT/US96/15465 



9 /11 




SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



10/11 



PCT/US96/15465 



710 

i_ 



720b 



720c 



720o 

7~ 



PARAMETER 
BUFFER 



I 



702 



700 



MPEG STREAM SELECTOR 
AND 
MULTIPLEXER 



I 



730 

2. 



r 740 



L 



704 



SUBSCRBER PARAMETERS/ 
MPEG STREAM CONTENT BREAKS 



FIG. 10 



SUBSTITUTE SHEET (RULE 26) 



PCT/US96/15465 



11/11 



( START 



4000 



COPY HEADER OF 
PRIMARY VIDEO STREAM 
INTO PARAMETER BUFFER 



I 



DELIVER HEADER OF 
PRIMARY VIDEO STREAM 
TO OUTPUT CHANNEL 



.4002 



>004 



FIG. II 



DETERMINE NEXT BREAK POINT 
FOR DELIVERING SECONDARY 
VIDEO STREAM 



1 



SEND PRIMARY 
VIDEO STREAM TO 
OUTPUT CHANNEL 



STOP PRIMARY 
VIDEO STREAM 



SEND RESET CODE 
TO SET-TOP UNIT 



SEND SECONDARY 
VIDEO STREAM OT 
OUTPUT CHANNEL 



4006 



4008 




~1 



( END ) 



4024 



,4014 



*4016 



.4018 



EOF SECONDARY 
VIDEO STREAM? 



4020 



GENERATE NEW HEADER 

COPY HEADER OF 
PRIMARY VIDEO STREAM 
INTO PARAMETER BUFFER 



4022 



SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 



AamtM v> ******* »»*m CUsifictton «PO or «° *>* cUnlfiaaa0 
B. MBI.IK S EARCHED 

IPC 6 HG4N 



lata <oul AppdcHico No 

PCT/US 96/15465 



I Electronic data base 



consulted during the intemaoonal search (name 



of data base and, where practical, search terms used) 



1 r DOCUMENTS CONSIDERED TO BE RELEVANT 



P.X 



Citation of document, with indication, where appropriate, of the relevant passages 

EP,A,0 656 728 (SONY CORP) 7 June 1995 

see page 3, column 3, line 3 - line 38 
see page 4 f column 5, line 43 - page 6, 
column 9, line 31 
see figures 1-4,9-11 

EP,A,0 680 214 (AT & T CORP) 2 Novetrter 
1995 

see page 2, column 1, line 51 - column 2, 

line 32 „ . c 

see page 4 t column 5, line 3 - page b, 
column 8, line 19 
see figures 3-5 

-/-- 



Relevant to daira No. 

1-3,5-8, 
10,11 



1-3,5-8, 
10,11 



I p^j Further documents are lifted m the continuation of box C. 

• Special catefones of a ted document* : 

•A* document defimnf the fcneral state of the art which a not 

considered to be of particular relevance 
- E - e^jiier document but published on or after UK international 

filing date 

"i" document wtnch may throw doubts on pnomy dai™(i)of 
wSchTated to establish the pubucatton date of another 
citation or other speaal reason (as spea fied) 

-O" document referring to an oral dudosure, use, odutabon or 
other means 

"P* document published pnor to the international fUini date but 
later than the priority date claimed 

I Date of the ictual completion of the internanonal search 



Patent family members are listed in I 



-r law document published after the »trrnanor^&un»date 
a^pnonrTdate and not in conflict with the «P^canon but 
^eTto^undottand the principle or theory underlying the 



10 January 1997 



Name and mailing address of the ISA 

European Patent Office. P.B. SS18 Patentlaan 2 
NL • 22S0 HV Rijswijk 
Td. (+ 31-70) 340-2040, Tx. 31 651 epo nl, 
Fax: 31-70) 340-3016 



document of particular relevance; the dain^ mvennon 
Sootbe oonndered novd or cannot be considered to 

document of particular relevance; the dainKd invenoccri 
^STbe co£dertd to involve an inverinj^ wh«Ae 
Surnent is combined with one or more ****** *£\ 
SSTsuch combination being obvious to • P«on stalled 
in the art. 

' document member of the same patent family 
Date of mailing of the internanonal search report 



28. 01. 97 



Authorized officer 



Fore PCT,1SA/J10 ( 



Van der Zaal, R 



page 1 of 2 





Into uofial Application No 1 


| C^ConOnuanog) DOCUMENTS CONSIDERED TO BE RELEVANT ' "~ 


PCT/US 96/15465 



Ca*g«y ' I Citato, of document «th Jetton. where .pprojmue, ,f the relent pisagc,' 



15654 (ZING SYSTEMS L P) 8 June 

1995 

see page 5, line 10 - page 7, line 8 
see page 15, line 11 - page 16, line 14 
see page 17, line 10 - line 22 
see figures 1,15 

EP.A.G 536 628 (GEN INSTRUMENT CORP) 14 
April 1993 ' 
see page 3, line 37 - page 4, line 3 
see page 5, line 27 - line 52 
see figure 2 



Relevant to ctum No. 

9-11 



1.3,5-8, 
10,11 



Foim PCT TSA 210 {continu»t>on of second sheet) (July it«> 



page 2 of 2 



INTERNATIONAL SEARCH REPORT 

Information on patent tally manbas 



Patent document 
cited in search report 

EP-A-0656728 



Publication 
date 

07^06^95 



Into *ooal Application No 

PCT/US 96/15465 



Patent family 
member(s) 



JP-A- 
CN-A- 



7233420 
1115948 



Publication 
date 

04-08^95 

31-01-96 



EP-A-0680214 


02-11-95 


US-A- 
CA-A- 
JP-A- 


5532735 
2145565 
7303087 


30-10-95 
14-11-95 


W0-A-9515654 


08-06-95 


AU-A- 
EP-A- 


1Z92895 
0740884 


06-11-96 


EP-A-0536628 


14-04-93 


US-A- 


5231494 


27-07-93 



form PCT/ISA210 (pstcnt family mum) (Juty 1992) 



THIS PAGE BLANK (uspto) 



( 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 



U BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




THIS PAGE BLANK <uspto> 



